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Introduction 



• UAS in the NAS Project 

- Developed to help address technical barriers to integration of unmanned aircraft 
systems (UAS) into the national airspace system (NAS) 

• Findings help guide development of RTCA Special Committee 228's Minimum 
Operational Performance Standards (MOPS) for UAS 

- An outstanding technical barrier is providing UAS pilots with a means to "detect 
and avoid" other aircraft 

• Means of compliance with 14CFR, Section 91.113 - pilots must remain well clear from 
other aircraft through "see and avoid" 

• Detect and Avoid (DAA) System 

- A collection of technologies - consisting of both hardware & software - that can 
provide pilots with the necessary information to self-separate from other aircraft 

- A traffic display would serve as substitute for manned pilots' ability to see outside 
their aircraft 

- Critical question: what are the display requirements for such a system? 
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Introduction 



• DAA Display Research 

- Several part-task studies have looked UAS traffic display 

• Friedman-Berg et al. (2014) & Draper et al. (2014) focused on identifying the minimum 
information requirements 

- Results were largely in agreement, most significant difference being the inclusion of maneuver 
recommendations (display guidance) in Draper et al. 

• Bell (2012) found that "advanced" displays - i.e., those that provided a level of display 
guidance - led to less severe separation violations than displays without guidance 


Intruder Information 

Friedman-Berg 
et al. (2014) 

Draper et al. 
(2014) 

Bell (2012) 
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Introduction 



• Current Study: 

— Continues examination of DAA display requirements within a "full mission" task 
environment 

• Tests different information levels (no guidance vs. guidance) and location of the traffic 
information 

- Standalone (i.e., bootstrap) displays may be easier to develop, but may hinder performance 
relative to displays integrated into command and control interface 

• Focus on the displays' impact on pilots' measured response (MR) 

- MR can be understood as the quantification of the end-to-end response time for a 
UAS pilot to complete a self separation maneuver in response to a DAA display 
alert 

• Measured response metrics can reveal the amount of time pilots spent interacting with 
different displays, allowing direct comparisons 

• Longer MR times may result in delayed maneuvering, which can in turn increase 
likelihood of a separation violation 

• MR times can also inform human response models that are used in fast-time simulation 
and the alerting threshold parameters used by the alerting logic 


4 


Method 


• Participants 

- 12 active UAS pilots (M = 39 years of age) 

• All had military UAS experience (avg. 216 hrs) 

• 8/12 had civil UAS experience (avg. 60 hrs) 

• Simulation Environment 

- Ground Control Station 

• Vigilant Spirit Control Station (VSCS; right) 

- Provided command and control interfaces, 
aircraft information, and a simulated out-the- 
window view, across 3 monitors 

— Displayed traffic information in select 
conditions 

- Mouse and keyboard inputs only 

• Cockpit Situation Display (CSD; right) 

- Standalone CDTI only active in select 
conditions 

- One monitor, directly to left of VSCS monitors 



Cockpit Situation Display 
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Method 



• 2x2 Repeated Measures Experimental Design 

— Information Level: 

• Basic Information 

- Standard intruder information (as set by Friedman-Berg et al., 2014) 

» No display guidance 
— Multi-level alerting 

• Advanced Information 

- Standard intruder information 

- Multi-level alerting 

» Included additional level 

- Suite of guidance tools 

» Trial planning tools 
» Recommended Maneuvers 

- Display Location: 

• Standalone Display 

- Information presented within CSD 

- Pilots still used VSCS to input changes to aircraft 

• Integrated Display 

- Information presented within VSCS command and control interface 
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Method 


Multi-Level Alerting 

- Visual and auditory alerts tied to predicted threat level of nearby traffic 

- Based on predicted closest point of approach (CPA) between ownship and intruder 

• Horizontal miss distance (HMD), vertical miss distance (ZTHR), and time to CPA criteria 
all had to be satisfied to be assigned given threat level 

- Pilots instructed to maneuver prior to collision avoidance alert being generated, 
which was their indication that separation had been lost 


Alert/Threat Level 

HMD 

ZTHR 

Time to CPA 

Symbol 

Proximal 

> 2 NM 

> 900 FT 

N/A 

A 

Preventative 

< 2 NM 

< 900 FT 

< 120 secs 

A 

Self Separation 

< 1.2 NM 

< 900 FT 

< 110 secs 

A 

Predicted CA Alert* 

<0.8 NM 

< 400 FT 

< 110 secs 

A 

Collision Avoidance 

<0.8 NM 

< 400 FT 

< 40 secs 

A 


*Only present in the Advanced Information display conditions 







Method 


1. Basic Standalone Display 

— Standard intruder information and multi-level alerting presented within CSD 

• No display guidance provided 

• VSCS served as command-and-control interface 
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Method 


2. Basic Integrated Display 

— Standard intruder information and multi-level alerting presented within VSCS 

• No display guidance provided 

• Traffic info collocated with vehicle control interfaces 
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Method 


3. Advanced Standalone Display 

- Display guidance included, in addition to standard info and alerting: 

• Trial planning tools allowed pilots to test different heading/altitude maneuvers before 
uploading 

• Maneuver recommendations offered suggested solutions 



Method 


4. Advanced Integrated Display 

- Display guidance included, in addition to standard info and alerting 

• Trial planning tools allowed pilots to test different heading/altitude maneuvers before 
uploading 

• Maneuver recommendations offered suggested solutions 
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Method 



• Pilot Task 

— Operate simulated MQ-9 Reaper within civil airspace, under Instrument Flight 
Rules 

• Routes contained entirely within Class E, Oakland Center airspace 

- Instructed to coordinate maneuvers with ATC (over push-to-talk headset) 

• Missions lasted 40 minutes 

— Maintain well clear from nearby aircraft 

• 8 scripted encounters with the ownship (i.e., would lose separation absent of pilot 
intervention) 

• Additional tracks were included to emulate busy day at Oakland Center 

- Attend to secondary tasks 

• Respond to requests for status information (e.g., current fuel level 

• Complete electronic checklists in response to system malfunctions 
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Method 


Confederate Participants 

— Retired ATC managed all aircraft within experimental airspace 

— "Pseudo" pilots controlled simulated manned aircraft within airspace 

A researcher coordinated in real-time to ensure conflicts were generated 





Method 



• Measured Response Metrics 

— A pilot-DAA interaction timeline was constructed, with emphasis on the pilots' 
interaction with ATC and the GCS (below) 

• The timestamps for each stage of the timeline were collected from a variety of sources: 

- GCS output files, DAA algorithm output files, voice recordings and logs, and video 
recordings 


Stage 

Description 

T 0 

DAA (self separation or collision avoidance) alert 
appears on the display 

Ti 

Pilot notifies ATC and requests a maneuver clearance 

t 2 

ATC provides maneuver clearance 

t 3 

Pilot initiates an edit in GCS to maneuver 

t 4 . 

Pilot uploads 1st maneuver to aircraft 

"^4b 

Pilot uploads final maneuver to aircraft 
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Method 



Measured Response Metrics 

- From these timestamps, five metrics were extrapolated: 

• Total Response Time (T 4b - T 0 ) - how long it took the pilot to upload an appropriate 
maneuver following a DAA alert 

• Initial Response Time (T 3 -T 0 ) - how long it took the pilot to initiate an edit in the GCS 

• Total Edit Time (T 4b - T 3 ) - how long it took the pilot to implement appropriate 
maneuver 

- Initial Edit Time (T 4a - T 3 ) - how long it took the pilot to implement an initial 
maneuver 

• Notification Time (T^Tq) - how long it took the pilot to notify ATC following an alert 
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Results 



Measured Response Metrics 

- From these timestamps, five metrics were extrapolated: 

• Total Response Time (T 4b - T 0 ) - how long it took the pilot to upload an appropriate 
maneuver following a DAA alert 

• Initial Response Time (T 3 -T 0 ) - how long it took the pilot to initiate an edit in the GCS 

• Total Edit Time (T 4b - T 3 ) - how long it took the pilot to implement appropriate 
maneuver 

- Initial Edit Time (T 4a - T 3 ) - how long it took the pilot to implement an initial 
maneuver 

• Notification Time (T^Tq) - how long it took the pilot to notify ATC following an alert 
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Results 



• Total Response Time 

— Significant main effect of Information Level on Total Response Times (p < .05) 

• Pilots took an average of 37.87s to complete their final edit in response to SS/CA alerts 
(from first alert appearance) 

- Pilots 8s faster (19%) on average in Advanced than Basic conditions 

— No other significant main effects or interaction 
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Results 



Measured Response Metrics 

- From these timestamps, five metrics were extrapolated: 

• Total Response Time (T 4b - T 0 ) - how long it took the pilot to upload an appropriate 
maneuver following a DAA alert 

• Initial Response Time (T 3 -T 0 ) - how long it took the pilot to initiate an edit in the GCS 

• Total Edit Time (T 4b - T 3 ) - how long it took the pilot to implement appropriate 
maneuver 

- Initial Edit Time (T 4a - T 3 ) - how long it took the pilot to implement an initial 
maneuver 

• Notification Time (T^Tq) - how long it took the pilot to notify ATC following an alert 
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Results 



• Initial Response Time 

— Near significant effect of Display Location on Initial Response Times (p = .054) 
• Pilots took an average of 19.32s to initiate an edit in response to a SS/CA alert 
- Pilots 5s faster (23%) in Standalone display conditions 
- No other significant main effects or interaction 
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Results 



Measured Response Metrics 

- From these timestamps, five metrics were extrapolated: 

• Total Response Time (T 4b - T 0 ) - how long it took the pilot to upload an appropriate 
maneuver following a DAA alert 

• Initial Response Time (T 3 -T 0 ) - how long it took the pilot to initiate an edit in the GCS 

• Total Edit Time (T 4b - T 3 ) - how long it took the pilot to implement appropriate 
maneuver 

- Initial Edit Time (T 4a - T 3 ) - how long it took the pilot to implement an initial 
maneuver 

• Notification Time (T^Tq) - how long it took the pilot to notify ATC following an alert 
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Results 



• Total Edit Time 

— Significant main effect of Information Level on Total Edit Times (p < .01) 

• Pilots took an average of 17.65s to complete their final edit in response to SS/CA alerts 
- Pilots 9s faster (40%) in Advanced display conditions 
- No other significant main effects or interaction 
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Results 



Measured Response Metrics 

- From these timestamps, five metrics were extrapolated: 

• Total Response Time (T 4b - T 0 ) - how long it took the pilot to upload an appropriate 
maneuver following a DAA alert 

• Initial Response Time (T 3 -T 0 ) - how long it took the pilot to initiate an edit in the GCS 

• Total Edit Time (T 4b - T 3 ) - how long it took the pilot to implement appropriate 
maneuver 

- Initial Edit Time (T 4a - T 3 ) - how long it took the pilot to implement an initial 
maneuver 

• Notification Time (T^Tq) - how long it took the pilot to notify ATC following an alert 
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Results 



Initial Edit Time 

— Significant interaction between Information Level and Display Location on Initial 
Edit Times (p < .01) 

• Pilots took an average of 11.77s to complete their first edit in response to SS/CA alerts 

- Difference between Basic and Advanced displays in Integrated conditions was 12s 
(68%), while only 2.5s (12%) in Standalone conditions 

- Information Level had a significant main effect (p < .05), 6.5s faster in Advanced 
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Results 



Measured Response Metrics 

- From these timestamps, five metrics were extrapolated: 

• Total Response Time (T 4b - T 0 ) - how long it took the pilot to upload an appropriate 
maneuver following a DAA alert 

• Initial Response Time (T 3 -T 0 ) - how long it took the pilot to initiate an edit in the GCS 

• Total Edit Time (T 4b - T 3 ) - how long it took the pilot to implement appropriate 
maneuver 

- Initial Edit Time (T 4a - T 3 ) - how long it took the pilot to implement an initial 
maneuver 

• Notification Time (T^Tq) - how long it took the pilot to notify ATC following an alert 
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Results 



• Notification Time 

— Near main effect of Information Level on Notification Times (p = .059) 

• Pilots took an average of 29.07s to notify ATC of a maneuver in response to a SS/CA 
alert 

- Pilots 6s faster (19%) in the Advanced information conditions 

— No other significant main effects or interaction 
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Results 



• Summary 

— Advanced Information displays showed advantage in four of five reported metrics 

• Total Response Times 19% shorter in Advanced conditions 

• Total Edit Times 40% shorter in Advanced conditions 

• Initial Edit Times 70% shorter in Advanced Integrated condition than in the Basic 
Integrated condition 

• Notification Times 20% shorter in Advanced conditions 

- Overall benefit seen for lower Total Response Times was due to a reduction in how 
long pilots spent interacting with the display 

• Not how quickly they got 'in-the-loop' (Information Level did not impact Initial Response 
Times) 

- Display Location only approached significance in one of the metrics 

• Initial Response Times 23% shorter for Standalone display 
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Discussion 



• Advanced Information 

- The presence of display guidance (in a variety of forms) reduced the amount of 
work required of the UAS pilot 

• The Advanced displays unambiguously alerted pilots of which self separation threats 
were predicted to lose well clear 

• The tools provided the pilot with a pre-determined maneuver, limiting the amount of 
time they had to spend calculating their own 

- Led to pilots contacting ATC more quickly 

• However it was clear that pilots often initiated edits prior to contacting ATC 

- Roughly 50% of maneuvers occurred without prior ATC approval 

• Display Location 

- Did not have a significant impact on pilot performance 

- The lack of immediate pilot responses may have mitigated the lack of an effect of 
display location (Initial Response Times were on the order of 20s) 
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Conclusion 



• The first in a collection of studies, this sim demonstrated that Information 
Level, namely the absence or presence of display guidance, can substantially 
impact pilots' response times 

- Future studies have been conducted that look at different sorts of display guidance 
to see if certain implementations result in superior performance 

• This data is supplemental to other objective metrics - mainly rates of 
separation violations and pilot feedback - but supports inclusion of display 
guidance 

- Santiago and Mueller (2015) - found 45% fewer losses of well clear when pilots 
were provided with display guidance 

• Faster pilot responses is one reason for the finding, among less ambiguous alerting and 
eliminating the need for the pilot to self-determine a maneuver 

• Quick pilot inputs were especially important in cases of 'pop-up' encounters, where 
there was a small amount of time before a loss of well clear would appear 

- Monk et al. (2015) - found pilots preferred the Advanced displays 

• Supported more immediate responses 

• While all displays were rated as sufficient, Advanced Integrated was rated as most 
preferable 
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Conclusion 



• Limitations 

- Cannot necessarily generalize to other GCS 

• Different GCS have different vehicle control inputs 

• There was a high level of integration between the Advanced features and the GCS in the 
Advanced Integrated condition 

- It is possible to present display guidance with less integration, which may impact 
pilot performance 

— There were multiple feature changes between the Basic and Advanced conditions 

• Several tools were included, as were several advanced pieces of information, including a 
new alerting level 

• 2 follow-on studies have been submitted to different conferences 

- Both look at different ways to provide display guidance in an integrated fashion 

- HITL data to be validated in flight test environment 
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